Incident Report: Persisting Genuine Stuck Collector Alert
Date: 2023-09-30
Time: 23:49 (GMT+5:30)
Duration: 01:15 hours:minutes
Description
Persisting Genuine Stuck Collector Alert on fantom (on staging as well as production): Detected stuck collector on fantom: current block (68778020) and last queried block (68777528) recorded on 2023-09-29T18:02:46.085.
Root Cause
Problem with the deployment happened earlier that day.
Impact
Low impact on devs.
Timeline
- 23:50 - First noticed the issue.
- 12:00 - Initial diagnosis.
- 12:40 - Started the fix.
- 01:02 - Issue resolved.
Lessons Learned
Deployments needs to be checked and properly monitored during the fixes.
Actions Taken
Aaron redeployed event collectors.
Related Images/Logs
Incident Reviewer(s)
- Andrew
- Aaron